feat(backup): RestoreTableFromBackup job and cross-backend matrix [backup/restore 9] - #740

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job · this PR: the restore job from the store, every backend, the cross-backend matrix and extenddb backup remove-target (the design's PR 9a and PR 9b as one PR, with PR 10's remove-target pulled forward) · next: PR 10 backup sync.
Closes #739 (part of #624). Fixes #742 (the Cassandra write path deleting the index rows of an unchanged item).

Stacked on #700 (feat/backup-job, head 09c881f4). This PR adds restore_progress to the PostgreSQL data database (migration 6); docs/schema/postgres.txt gains the table, sqlite.txt is unchanged, both checked against a fresh init. It binds PR 7's store, spool and health and PR 6's claim, row and reconciler to the restore: check_catalog_row_for_restore no longer refuses a store-backed backup; the request takes require_healthy_store and the job permit and registers the target under the backup row's lock; the reconciler's cycle gains the dead-restore resume and the restore_progress and spool sweeps beside PR 8's backup duties.

What

Core (crates/core). No new rule; validate_restorable_item, RestorableLimits and the manifest reader (ExtenddbManifest::reader, reader_prehashed, the per-file KeyRange check) are PR 4's and PR 8's, bound by the restore. DataFileDecoder and DataFileReader expose bytes_decoded(), the uncompressed bytes decoded so far, which the load's chunk bound counts.

Storage trait (crates/storage). A new RestoreEngine trait in StorageEngine's bound: bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> applied, finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }, count_restore_target, read_restore_progress (default []), sweep_restore_progress (default 0); RESTORE_BATCH_ITEMS (10,000) and RESTORE_BATCH_BYTES (32 MiB), the SQLite and Cassandra batch bounds, live beside it. BackupCatalog gains claim_restore(account_id, input, &RestoreRegistrationRow) -> (TableDescription, Box<dyn JobClaim>) (the registration creates the CREATING target and the RUNNING record under the backend's own generation), claim_restore_dead(key, stale_after, spool_host) (the restore analogue of claim_dead_backup; it writes a fresh heartbeat_at on the database's clock in the same conditional write as the generation and spool_host, so a second taker finds the record live at once, where the backup takeover leaves the heartbeat for its FAILED-prefix clock), read_restore_job and list_restore_jobs; RestoreJobRow carries the store, spool_host, file_index, items_applied, failure_reason and completed_at. restore_input_from_manifest builds a CreateTableInput from the manifest's source_table and applies the overrides, one implementation for both restore paths. restore_pause_point(name) is a pub async fn under test-hooks, a no-op without it, and the four backend pause-point names are pub consts beside it (RESTORE_SQLITE_MID_BATCH, RESTORE_MONGODB_AFTER_INSERT_MANY, RESTORE_CASSANDRA_BEFORE_PROGRESS, RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE), one spelling each: the engine's KillPoint holds the engine's five points alone. JOB_HEARTBEAT_INTERVAL's doc says the store-backed jobs run at max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL) and the in-catalog paths and the maintenance claim at the constant.

PostgreSQL (crates/storage-postgres). restore_engine.rs: bulk_load opens one data transaction per file, multi-row INSERTs of 500 base and index rows (or 4 MiB of item text) through insert_restore_rows, writes the restore_progress row in the same transaction (a plain INSERT for a file applied from its start, an upsert on (table_id, file_index) for a file resumed past it, so a row a decoder panic left short is completed by the resume rather than failing on the row's key), and fails a duplicate on the base table; nothing in it conditions on the generation (the base table's key and the claim's liveness are its fence). finish_restore flips the table and every index ACTIVE in one catalog transaction and then deletes the progress rows on the data pool (the pause point restore_postgres_before_progress_delete between the two); that delete's error is logged and swept by the reconciler, never the restore's failure, since the table is ACTIVE and the record DONE once the flip commits. claim_restore locks backup_index FOR UPDATE (the lock start_backup_delete takes; its comment says so); claim_restore_dead takes the name's advisory lock and a stale RUNNING manifest record, refreshing heartbeat_at with the generation and spool_host. sweep_restore_progress deletes from a positive list (the table ids the progress rows name, read first, minus the RUNNING records read after and the caller's list), so a registration that commits between the two reads is never a candidate. Migration 6 adds restore_progress(table_id, file_index, items_applied). remove_restore_target's to_regclass('restore_progress') check and its DELETE now run on the data pool; on 09c881f both ran on the catalog pool, so with migration 6 in a separate data database the PR 6 code found no table and deleted nothing. remove_restore_target drops the vector index's _ddb_vec_ relation with the data and index tables (table_storage_relations); the record's heartbeat_at is the registration transaction's now(), refreshed by the engine's first beat before the reply.

SQLite (crates/storage-sqlite). bulk_load writes batches of at most 10,000 items or 32 MiB of serialized item text, each a BEGIN IMMEDIATE holding the base, index and vector rows and the record's (file_index, items_applied), conditioned on the generation (the pause point restore_sqlite_mid_batch between batches); claim_restore takes the name's slot; claim_restore_dead takes over a record whose slot no task holds, whatever its heartbeat (a restart frees every slot and ages every RUNNING record), refreshing heartbeat_at. The startup line for an aged RUNNING record names the reconciler's duty per kind (a store-backed restore is resumed; an in-catalog one is removed). remove_restore_target no longer probes for a restore_progress table (SQLite never has one; a store-backed restore's progress is on its table_restores row, file_index and items_applied). The flip's backfilling = NULL on the vector rows is now observed by a fixture that sets it before the flip. A table with secondary indexes restores in more resident memory than the no-index figures the docs quote, and the excess is SQLite's own per-connection page cache for the index b-trees plus the allocator's arenas, not the restore's buffers, which stay bounded by the batch's 32 MiB (the module doc, the admin guide and the record state the shape beside the figures; a smaller batch for an indexed table was built, measured ineffective and not kept).

MongoDB (crates/storage-mongodb). bulk_load writes insert_many(ordered=false) of 1,000 base documents at w: majority, reading the record before each call and ending the file as a lost claim when it is not RUNNING under the generation, suppressing an all-E11000 batch only when reapplying, then records restore_job.file_index (the pause point restore_mongodb_after_insert_many between); finish_restore writes DONE and restore_backfill_pending and returns Pending, the backfill worker flipping the table ACTIVE. claim_restore_dead refreshes heartbeat_at with $$NOW, so a second taker right after the first is refused; the registration stamps the record's heartbeat from the server's clock inside its transaction. An N sort key past Decimal128 fails the file with the write path's text.

Cassandra (crates/storage-cassandra). bulk_load upserts in batches of at most 10,000 items or 32 MiB of serialized item text, eight writes in flight, reading the record before each batch, and records progress with a lightweight transaction per batch (the pause point restore_cassandra_before_progress); finish_restore is one LWT (Active, never Pending; index rows created ACTIVE); count_restore_target checks the catalog first, as the other three do. The count is the only duplicate check. An unknown outcome of the progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore); a registration whose record was taken over during its DDL removes nothing and answers the lost claim; the test-hooks feature gains fail_progress_writes (Landed, Lost). claim_restore_dead refreshes restore_heartbeat_at in its LWT and re-applies the backup's active_restores entry for the target when a crash between the target's DDL and the registration left none, so the in-use rule holds through the resume. The write path rewrites an index row in place when a PutItem or UpdateItem leaves the index keys unchanged (#742: sync_indexes and the GSI worker's gsi_apply_index put the old row's DELETE and the new row's INSERT in one LOGGED BATCH, whose one timestamp let the tombstone win, so a re-put of an unchanged item deleted its GSI and LSI rows; one shared write_index_row_change now skips the DELETE when the old and new items give the same index row), so a re-applied restore batch keeps every GSI and LSI row where before #742 was corrected a resumed Cassandra restore reached ACTIVE with up to one batch of index rows missing. Two concurrent registrations of one backup leave one target: the registration's LWT (IF backup_status = 'AVAILABLE' AND active_restores = null) is the fence, linearized on the backup's partition; the in-use pre-read is advisory. New index-row tests: a synchronous re-put over a composite GSI+LSI target, the GSI worker's path, a re-applied bulk_load batch, and the two-registration race.

Engine (crates/engine/src/backup/). restore_table_from_backup and the reconciler take &Arc<S> (the manifest branch and the resume spawn tasks). restore.rs and restore/{spool,download,load}.rs: the request verifies the sidecar and manifest (each read bounded, 4 KiB and 16 MiB; an object over its bound is a ValidationException naming a corrupt prefix), the vector gate, the overrides, the key schema and the target, registers under timeout(RESTORE_REGISTRATION_TIMEOUT) (60 s, every backend), and runs the job (download with a 15-minute transport retry counted from the job's first attempt, per-file bulk_load, the count, finish_restore). The load's chunk bound (LOAD_CHUNK_BYTES, 32 MiB) counts the decoded line's bytes, and the feeder checks the claim at every chunk boundary beside the shutdown token; a decoder that dies fails the file. .verified markers hold the file's sha256 and a stale entry (a record that was another host's) is removed before use; a job whose record is another's (a lost claim, a DONE or FAILED write that matched no record) leaves its spool entry to the sweep. The spool sweep lists the spool before the records and keeps the entry of a restore registered between the two reads; the spooled manifest and sidecar are written .partial → sync → rename and a spooled pair that fails verification yields to the store's; a finish_restore error is re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); the resumed job's first beat is the takeover's return, taken by the reconciler beside claim_restore_dead; BackupStoreErrors{get} on every get the store failed; the restored table's size is the manifest's source_table.table_size_bytes; a registration that fails after the request's 60 s bound is logged with the ARN, the target and the account. The job sends its first heartbeat right after claim_restore returns and before the reply, and claimed_at is that beat's return, so a slow registration does not end the job at its first check and the record's heartbeat_at is fresh at the reply (on PostgreSQL now() is the transaction's start, so a registration stalled on the backup_index lock or the DDL would otherwise commit a stale stamp); a first beat that finds the claim lost fails the record ("the claim was lost before the job started") and answers InternalServerError; the maintenance refusal records RestoreJobs{failed}. remove-target refuses or removes a RUNNING record's target by list_dead_restore_jobs on the database's clock (no host clock), and the purge's wait reads the backend's heartbeat listing the same way; the reconciler takes nothing over once the shutdown began and applies the target_not_empty guard under a maintenance claim (a dead record with no progress over a target that holds items is failed target_not_empty and left for remove-target, not OWNER_DEAD and removed); the lost-claim stop line carries the heartbeat timer's reason, which the timer also logs at WARN; the spool entry is removed on a failed spool document write and on a resume that fails before it runs; a DONE write whose answer was lost and whose record re-reads DONE ends with complete (DONE); the two target_not_empty lines print extenddb backup remove-target --account <account> <table>. The reconciler checks the store first in its cycle (a store whose first check has not recorded is not yet checked, not unhealthy), resumes a dead store-backed restore from its progress under claim_restore_dead, fails one of another store (OTHER_STORE) or no progress into a non-empty target (target_not_empty), counts a maintenance-held dead record in a field of its own, and sweeps restore_progress rows and spool entries. Metrics RestoreJobs, RestoreJobDuration, RestoreBytesDownloaded (each data-file download attempt's bytes); KillPoint gains five restore variants (RestoreAfterDownload, RestoreAfterFileCommit, RestoreMongodbBeforeWorkerFlip, RestoreBetweenFailedAndRemoval, RestoreFencedOut; the engine's list; the engine's points log job reached kill point <name> and the backends' restore job reached kill point <name>). crates/server/Cargo.toml's backup-test-hooks adds extenddb-storage/test-hooks so the backends' pause points compile in.

App (crates/app). extenddb backup remove-target --account <id> <table> (PR 10's command, pulled forward): under the maintenance claim it removes a FAILED restore's CREATING target and record (target_not_empty included, the state an ordinary crash before the first progress record leaves on MongoDB and Cassandra), a RUNNING restore's target whose owner is dead, or a legacy record's target, through remove_restore_target; it refuses a live RUNNING restore, a DONE restore and a table no record names. It needs no store, prints the claim it took and one line naming what it removed, and logs the removal under extenddb::audit. The command's logic and its tests are the engine's (maintenance::remove_target).

Harness and CI. The 24 restore cells of tests/test_backup_restore_fidelity.py run on a server with a store, and test_create_backup_of_a_vector_table_is_refused is test_vector_table_backup_refused_in_catalog_and_restored_from_store, since it asserts the round trip with SearchVectors on a server with a store; tests/test_auth_permissions.py's restore cell and tests/rust's backup suite run against the store; tests/test_backup_job.py gains the tamper, SIGTERM, store-outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments (the multi-part cell enables allow_multipart_table_keys on its own deployment and compares the restored item set with the source's Scan; the hand-built prefixes rewrite the two DynamoDB export manifests too, so they are whole backups) and, on SQLite, the backup remove-target cell: a finished restore is seeded at the catalog back into the target_not_empty state (a RUNNING manifest record with no progress over a CREATING target that holds items, the shape a crash inside the first file on MongoDB or the first batch on Cassandra leaves), the restarted server's reconciler fails it target_not_empty and DeleteTable, DeleteBackup and a second restore are refused, the command removes the target and its record with the documented output, and the restarted server answers ResourceNotFoundException for the name, DeleteBackup succeeds and a new restore runs; tests/test_mongodb_backup_restore.py gains the vector-manifest refusal, the 35-digit Decimal128 failure and the 38-digit intact cells, and its hand-built prefixes rewrite the two export manifests as the job suite's do; devtools/run-mongodb-tests serves a filesystem store and exports its root, its log and EXTENDDB_TEST_BACKUP_STORE, and its --backup-store none serves no store for the legacy LSI fallback cell (the one wire cell of the in-catalog restore path on MongoDB, which no CI job runs; the test guide and the cell's docstring give the command); the runner starts from an empty store root, since a reused --output directory's extenddb-store.json names the previous run's catalog. The PostgreSQL CI job starts moto, so the store-outage cell runs there as on the SQLite S3 job, whose workflow comments now say that cell is the one S3 restore the job runs. The matrix is devtools/backup-restore-matrix and .github/workflows/matrix-nightly.yml: one target directory and four sequential builds, the artifact action pinned, one run at a time, requirements-matrix.txt for pymongo and cassandra-driver, an apt-get install postgresql-client step for the host psql the script's PostgreSQL seed and teardown call, and a step that checks psql --version and the two drivers' import before the cells; the SQLite Rust-integration job's comment in integration.yml says the backup tests restore the store-backed backups (it had said they expect the restore's refusal); its duplicate-key prefix writer chmods its files 0600 as copy_prefix does, and its docstring's exit codes are the script's (1 on an assertion failure or a setup error).

Docs. The differences document (vector, multi-part-key, how-restore-runs with its timings labelled as the in-catalog restore's beside the store-backed figures, concurrent-restore lock with Cassandra's two-registration guarantee, restore-interruption resume with the target_not_empty state, its remedy and the measured first-reach outcomes, a restored table's TableSizeBytes, a new Decimal128 row), the limits document (concurrent-restore spool and batch bound and Cassandra COUNT, backup contents), the usage guide and upgrade manual (the restore lands this release; migrations 1 to 5 breaking and 6 declared so for want of the additive-declaration job; the rollback paragraph; backup remove-target for the target_not_empty state), the admin guide (the spool's contents and sweep, the restore's memory terms and measured figures with the shape and the input order they were measured on, the two-GSI and the unordered figures beside them, backup list lists no restore, the backup remove-target command with its output and refusals, the owner's liveness on the database's clock, and the [backup.store] paragraph no longer saying a store-backed restore is a later release's), the developer test guide and CONTRIBUTING (the job suite, the MongoDB cells' variables and the runner's --backup-store none, the moto cells' credentials, the matrix and the nightly), docs/design/15-backup-restore-bench-pr9.md (the restore memory and duration record, with the two-GSI and the store-backed 60,000-item figures) and design revision item 30 with the body sentences it corrects.

Why

#396 section 5.3 and the PR 9a and 9b sections. PR 9a and 9b ship as one PR by the maintainer's decision (item 30). Decisions, each in the module doc of the code it concerns and in item 30: bulk_load is a RestoreEngine method, not a StorageEngine one; progress is recorded inside bulk_load in the write that applied the items, so there is no record_progress and PostgreSQL's progress row (in the data database, no job record) is fenced by the base table's key and the claim's liveness rather than the generation; reapplying makes MongoDB's duplicate suppression explicit; the manifest registration locks backup_index (the lock start_backup_delete takes) so a store-backed restore and a DeleteBackup of its backup order wholly; RESTORE_REGISTRATION_TIMEOUT is 60 s on every backend; a filesystem store is verified in place (two get passes), an S3 store spooled; the restore takeover refreshes the heartbeat where the backup takeover leaves it; the memory bounds are in bytes of text (the chunk in line bytes, the SQLite and Cassandra batch in serialized item bytes); the download retry bound is the job's; the matrix seed stands in for backup sync --adopt-store until PR 10. extenddb backup remove-target ships here rather than in PR 10 because the target_not_empty rule, written for a catalog restored from a dump, has as its common trigger on MongoDB and Cassandra an ordinary crash before the first progress record (both record progress after the write), and without the command this release had no way to remove such a target. #742 is fixed in the write path for every caller rather than in the restore alone, since the same delete-plus-insert deletes the index rows of any unchanged item a client re-puts. The spool sweep lists the spool before the records, as the progress sweep lists its rows first, so neither removes what a registration between its two reads created. A finish_restore error is an unknown outcome re-read through the claim, as the status writes are, since on PostgreSQL the reachable case (the progress delete after the flip's commit) is a finished restore.

Testing done

Counts are of the whole tree on #700's head 09c881f4; "new" is against it. Every command below ran on the final tree, except where the sentence says a count is an earlier run's of this PR.

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings, the same --features extenddb-engine/backup-test-hooks, and -p extenddb-storage-cassandra --features test-hooks --all-targets all clean; cargo check --locked -p extenddb-storage --lib with and without test-hooks clean; cargo check --locked --workspace --all-targets clean.
• extenddb-engine 336 passed (62 new test attributes against 09c881f, one removed, 61 net; 7 of them this pass: the maintenance-held target_not_empty guard, no takeover once the shutdown began, the loss raised inside a step named by the next check, the first beat refreshing the registration's heartbeat before the reply, the request stopping at the 60 s bound with the task going on, a resume that fails before it runs removing its spool entry, a spool document write failure removing the entry; the 28 of the previous pass: the slow registration, the stopped timer before each write and before a download attempt, the DONE write's NotMatched, the lost-claim entry left, Pending, the job-wide retry bound, a decoder that dies, the same-host spool resume, a marker with another digest, a crash's spool states, a file replaced between the passes (the replacement matching the original in compressed length, uncompressed length and item count and failing on the sha256 alone), a file over one chunk by bytes, the feeder's claim check, an unhealthy or unchecked store with a known identity, SSE on both paths, a restore registered between the sweep's two reads, a failed DONE write re-read, the resume's first beat, a truncated spooled manifest, a resume whose store documents fail verification, Progress::from_records on each backend's shape, a record gone after the takeover, a registration that fails after the request stopped waiting, the bounded document reads, remove_target over a FAILED, a dead RUNNING, a live RUNNING, a DONE and a legacy record); extenddb-core 728, extenddb-server 246, extenddb-app 44. extenddb-storage 200. extenddb-storage-postgres with the PG env: 353 passed over every binary before this round's two cells were added, and this round restore_job.rs 10 passed (the takeover test asserting the refreshed heartbeat, the progress delete failing after the flip, the resumed file's upsert) with every other binary clean up to the slow table_snapshot.rs, which the run's timeout cut short without a failure. extenddb-storage-sqlite: lib and tests 245 passed (restore_job_tests 10, a_batch_closes_at_the_byte_bound_before_the_count_cap new, the flip test setting backfilling before the flip). extenddb-storage-mongodb: unit 182 (the lib binary's 144 rerun this round; no MongoDB code changed in it), --lib -- --ignored live_ 43 in the earlier run (37 backup live_ cells, live_bulk_load_stops_at_a_lost_claim_between_two_insert_many_calls new), and the test-hooks fence test a_takeover_during_bulk_load_fences_the_file_index_write (--features test-hooks -- --ignored --test-threads=1) 1; the 43 live_ cells pass under the shared replica set with --test-threads=1: run in parallel, three legacy-path copy cells can time out waiting for a GSI-backfill worker within the 90 s deadline (an environment limit, not a logic defect). extenddb-storage-cassandra: lib 78, --features test-hooks --test direct_integration 213 passed in the full run before this round's three cells were added (tests/direct/backup_engine.rs 48 tests, five new: the active_restores re-apply, the delete between the registration's pre-read and its fence, the composite-key resume with index rows and an empty file, the byte bound, the lost claim between two batches), and this round's three new cells and two changed ones ran twice and passed (tests/direct/backup_engine.rs: two concurrent registrations leaving one target, the re-applied batch keeping every index row over a composite GSI+LSI target; tests/direct/index.rs: an unchanged item's re-put keeping its GSI and LSI rows, the GSI worker keeping the row), the full binary not rerun end to end on the shared container; --test ttl_integration 25 passed, 1 ignored, test_ttl_enable_rejects_asynchronous_gsi the known pre-existing failure and test_backfill_cursor_resume passing alone.
• Mutants, each built by a targeted edit, caught at the assertion it targets and reverted by the reverse edit: 52 in the engine and core (22 of them this round: the shipped spool-sweep order, finish_restore's error as a failure, the resume's instant at its entry, the spooled documents written in place, the fallback to the store removed, no metric on a refused get, the unlogged registration failure, the Cancelled arm recording a worker error, a zero duration, the read bound removed, the in-place pass not hashing again, from_records ignoring the MongoDB and PostgreSQL shapes, the sidecar and ARN re-checks dropped, the gone record as an error, the compressed size as the table's, and four on remove_target: the DONE refusal skipped, every owner live, the claim dropped unreleased, a zero window not read as every owner dead), 21 across the storage trait and the four backends (five this round: the Cassandra index-row skip removed, the GSI worker bypassing the shared writer, PostgreSQL's progress delete propagated, the plain INSERT for a resumed file, the SQLite flip's backfilling clause dropped).
• The four release binaries (--features postgres|sqlite|mongodb|cassandra, -p extenddb), the four hooks binaries (<backend>,extenddb-app/backup-test-hooks) and the mongodb-test-hooks build built on the final tree; CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets clean. docs/schema/postgres.txt and sqlite.txt match a fresh init of the release builds byte for byte (no schema changed this round); cargo test -p extenddb-storage-postgres --test doc_version_literals passes with the record in the tree.
• Wire legs through .review/wire-pr8.sh on the PostgreSQL and SQLite release builds with a filesystem store: tests/test_backup_restore_fidelity.py test_auth_permissions.py (the 24 restore cells and the vector round trip with SearchVectors) — SQLite 100 passed, PostgreSQL 100 passed, no skip (the multi-part-key cell moved to the job suite); and tests/test_backup_store_config.py test_backup_job.py by name with EXTENDDB_BINARY and the moto variables on both builds — SQLite 25 passed, 3 skipped (the PostgreSQL-only store cells skipped; test_backup_job.py 14 of 14, the remove-target cell included), PostgreSQL 16 passed, 12 skipped (the SQLite-only store cells and the SQLite-only remove-target cell skipped; test_backup_job.py 13 of 14), the outage cell against moto on both. The multi-part-key cell runs on both (the restored item set equals the source's Scan; on SQLite the preview's item paths keep one item per first HASH key value in the source itself, so the set is 5 of the 60 items written). The MongoDB cells through devtools/run-mongodb-tests: the CI way (filesystem) tests/test_mongodb_backup_restore.py 5 passed, 1 skipped (the legacy LSI fallback cell skipped), and with --backup-store none 3 passed, 3 skipped (the legacy LSI fallback cell and the two throughput cells passed; the three store-backed cells skipped).
• The restore's memory and duration (docs/design/15-backup-restore-bench-pr9.md; the admin guide's max_concurrent_jobs row and the design's section 7 quote it): release binaries, filesystem store, VmHWM over idle from a fresh start, 24 items of 27.7 MB (the worst shape) restore with a peak of 848 MiB on SQLite and 778 MiB on PostgreSQL under the default allocator (848 and 1,013 MiB at 12 and 48 such items on SQLite) and 392 and 427 MiB under MALLOC_ARENA_MAX=1; 1,200 items of 350 KB in one data file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key and no secondary index, the backup's lines in key order as the server's own backups are, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same 1M items as an unordered backup (every MongoDB-sourced backup and every multi-part-key table) 144 to 164 MiB on SQLite in 66 to 90 s (scattered inserts dirty the whole base b-tree; the restore's buffers are the same) and 16.4 MiB on PostgreSQL in 17.5 s; at 3M rows SQLite 109 to 116 MiB key-ordered and 171 to 174 MiB unordered, the final COUNT(*)'s share 22 to 28 MiB at both sizes, bounded by the counting connection's cache_size (32 MiB) rather than growing with the table, as the record had said from one measurement. The same 200-byte items with two full-projection GSIs, measured this round on the head's and on the final binaries: SQLite 143, 225 and 143 MiB over idle at 100k in three runs (10.2 to 13.9 s to ACTIVE) and 165 and 172 MiB at 300k (51.7 and 51.9 s), PostgreSQL 29.3 and 22.9 MiB at 100k (9.9 and 6.2 s) and 30.8 and 33.5 MiB at 300k (19.7 and 21.4 s); the excess on SQLite is the backend's per-connection page cache for the index b-trees (cache_size, 32 MiB) plus the allocator's arenas, not the restore's buffers, and a smaller batch for indexed tables was measured ineffective (91.7 against 92.7 MiB) and not kept; every GSI's Scan count equalled the item count after ACTIVE. The 60,000-item shape the differences document's in-catalog timings were measured on, restored from the store on the final binaries: response 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB), ACTIVE after 0.67 s, 1.85 s and 3.74 s (the MongoDB job's DONE at 1.8 s, the backfill worker's flip after), 9.3, 38.3 and 30.1 MiB over idle; the differences document's in-catalog timings for that shape now carry their label. The record carries the terms the code bounds and the sizing sentence (about 450 MiB per running restore of large items under one arena, times max_concurrent_jobs).
• The cross-backend matrix from devtools/backup-restore-matrix, one invocation per cell, on the final binaries: the twelve plain cells (PostgreSQL, SQLite, MongoDB sources into all four targets) PASSED (items, description, GSI and LSI queries match the source; 10 to 44 s each, the Cassandra targets the slowest), with the seed's timestamps as the job writes them; the four --duplicate-key cells and the eight --vector cells PASSED on the final binaries too (the --duplicate-key cells into Cassandra by the count alone, restored 12 items, the manifest lists 13; the --vector cells into PostgreSQL and SQLite with SearchVectors answering, into MongoDB and Cassandra refused before any table). The nightly workflow has had no run: workflow_dispatch needs the workflow file on the default branch, so it cannot be dispatched until this PR merges, and its first run (the schedule's, or a dispatch after the merge) is the proof of the runner image. The workflow installs postgresql-client and checks psql --version and python3 -c "import pymongo, cassandra" before the cells, since the matrix script seeds and tears down its PostgreSQL deployments through the host's psql, which no other workflow uses.
• The restore kill points (kill-points.py, posted with this PR; every ACTIVE outcome asserts the base table's Scan count and the GSI by-g's Scan count both equal the item count), each killed or signalled at the point, restarted and asserted, on the final hooks binaries this round where the row says so and on the previous round's otherwise (nothing on those paths changed between the two):

Every cell above ran on the pushed head's hooks binaries (4c35319); the cells marked "rerun on the rebuilt binary" ran again on the binaries built after this round's last corrections (the first beat before the reply, the Cassandra progress re-read and registration fence, the PostgreSQL vector-relation drop), the pg_terminate_backend cell as reworked (the shipped cell slept 2 s, terminated and asserted the final count alone, which a 2.6 s restore passed without the case running). The Cassandra points ran against a freshly recreated single-node container with replication_factor = 1 (no GCInspector, StatusLogger or "No nodes available" line in its log after the runs). Two MongoDB cells (restore_between_failed_and_removal, sigterm) failed once at setup with ResourceNotFoundException on the fixture's first write when a second deployment on the same replica set dropped the fixed databases under them, and passed when rerun alone.

Not changed here

PR 10's backup sync --adopt-store, --move-store, --forget-backup and migrate-legacy; PR 11's CI finalization and documentation, with the MongoDB and Cassandra 1M restores and the vector cell still to be measured; PR 13; PR 14's drop of the in-catalog format. The matrix seeds the target's backup_index row by hand in place of backup sync --adopt-store. MongoDB's insert_many unit stays 1,000 documents with no byte bound of its own (the data file bounds it: bulk_load is per file). SQLite's restore memory for a table with indexes stays SQLite's page cache (cache_size) above the no-index figure; a smaller batch for indexed tables was measured ineffective and not kept.

Breaking changes

kind = "disabled" is no longer needed to keep a deployment able to restore the backups it takes. Migration 6 adds restore_progress to the PostgreSQL data database: run extenddb migrate before starting the new binary. An S3 store needs spool_path on PostgreSQL, MongoDB and Cassandra for restores. The restore of a store-backed backup is asynchronous on every backend with a 60 s registration bound, as on the service. On Cassandra a PutItem or UpdateItem of an item whose index keys do not change now keeps its GSI and LSI rows (#742); before, it deleted them.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com